The present application relates to data communication in computer storage technologies, and more particularly to improving wide port data communications.
Serial Attached Small Computer System Interface (SCSI), commonly referred to as SAS, is a point-to-point serial protocol that defines data movement to and from computer storage devices, and a protocol stack for storage entities interconnections. A SAS domain, forming a SAS topology, is typically formed by a multitude of SAS storage network elements which may include initiator SAS devices, such as a host bus adapter (HBA), a SAS expander, such as a SAS switch, and SAS end devices, such as hard disk drives in a JBOD (abbreviation for “just a bunch of disks”). As is known in the art, JBOD provides neither redundancy nor performance improvements. Hard drives may be handled independently as separate logical volumes, or they may be combined into a single logical volume using a volume manager. Each SAS device is identified with a universally unique SAS address. A SAS port can be a narrow port or a wide port. A narrow port has only one physical link (PHY), and a wide port has multiple PHYs.
A SAS expander facilitates communication between various SAS storage network elements by providing the necessary switching and routing functions. A SAS expander may also facilitate the connection between multiple SAS end devices and a single initiator port. A SAS expander often provides a number of SAS ports to enable communications between SAS devices.
FIG. 1 is a simplified block diagram of a computer storage system 100 using a simple SAS topology. System 100 is shown as including a single expander SAS switch 103 (which is a self-configuring expander, also shown as scfg), one HBA (host bus adapter) 101, one JBOD 105 including an expander 106 (which is also a scfg) and two disks 108 and 109, each with a different SAS address A/B/C/D/E. SAS links 00-09 are used to connect all SAS elements as illustrated. Wde port 131 connects HBA 101 to SAS switch 103 and includes SAS links 00, 01, 02, and 03. Wide port 132 connects SAS switch 103 to JBOD 105 and includes SAS links 04, 05, 06, and 07. FIG. 1 also shows two narrow port connections, i.e., disk connections SAS links 08 and 09.
On power up or link reset, SAS links 00-03 are brought up, expander 103 sends IDENTIFY address frame to HBA with the SAS address B on each of the link 00-03, HBA receives the IDENTIFY address frames on link 00-03 and finds that they contains the same SAS address of B. HBA considers itself as being directly connected to the expander 103 through the wide port defined by links 00-03. In the similar way, expander 103 considers itself as being directly connected to the HBA through the wide port that includes SAS link 00-03.
The HBA starts topology discovery towards expander B, it first sends an open request with destination SAS address of B on any one of SAS links 00-03 (for example, 00 is chosen), and expander B receives the open request and responds with an open accept. In thus way the connection between HBA and expander B is established, and SAS link 00 is dedicated for the communication between HBA and expander B. HBA sends management commands to expander B to perform discovery on the SAS devices directly attached to expander B. Expander B responds with the PHY status and reports that expander C is connected through the PHYs of SAS link 04-07. The HBA then tries to perform discovery on expander C. The HBA sends an open request with a destination SAS address C on any of the SAS link 00-03 (e.g., 00), expander B receives the open request and looks up the destination SAS address in the direct route table, and then propagates the open request to any of the SAS links 04-07(e.g., 04). Expander C receives the open request and determines that the destination SAS address is itself. Expander C then responds with an “open accept” through SAS link 04 where the open request is received. Expander B propagates the “open accept” back to the HBA on SAS link 00, and thus the SAS connection is established from HBA to Expander C using SAS links 00/04. Then HBA sends SAS management commands to expander C to discover the PHY status on expander C, and determines that two disks are connected to expander C. In this way, HBA completes the discovery process and have all the knowledge of the SAS devices in the SAS topology. HBA will use the SAS addresses discovered in the discovery process and IDENTIFY address frames when it tries to talk to the SAS device in the topology.
Expander B can also initiate the discovery process. It can send an open request with the destination SAS address C on any of the SAS links 04-07 (e.g., 04). Expander C receives the open request and responds with an “open accept,” so the connection between expanders B and C is established on SAS link 04. Expander B sends management commands to expander C and discovers the PHYs on expander C. Expander C responds and reports that disk D and disk E are attached. Expander B configures disk D and disk E SAS address in its route table, and completes the discovery. SAS expander C performs a similar discovery process as expander B, and configure the HBA's SAS address A in its own route table.
The expander's discovery process will only configure SAS addresses in the route table for the SAS devices that are not directly attached. The directly attached SAS addresses will be configured in the direct route table, when receiving the IDENTIFY address frame from the peer SAS device (the identification sequence). For example, expander B configures direct route table for SAS address A and SAS address C when SAS links 01-03/04-07 are up and IDENTIFY frames are received in the identification sequence. Expander B configures route entries for SAS address D and SAS address E in the discovery process. Expander C configures the route table for directly attached devices and non-directly attached devices in a similar way.
When all the identification sequence and discovery sequence on HBA/expander B/C are completed, the route table in the expander B/C are properly configured and the HBA has the knowledge of all the expander B, C and disk D, E. HBA will use the SAS address discovered to establish the connection when trying to talk to a disk/expander.
When the HBA tries to access disk D, it sends an open request with destination SAS address of D to any of the SAS link 00-03, (say, 00 is chosen). Expander B takes the SAS address received in the open request (SAS address D) and propagates it to any of the SAS link 04-07 (say, 04 is chosen). Expander C receives the open request and looks up the route table for SAS address D and propagates the open request to SAS link 08. The disk with SAS address D receives the open request and responds with open accept to SAS link 08. The open accept is propagated back to the HBA on SAS links 08->04->00, and thus the connection between the HBA and disk D is established. The HBA communicates with disk D through SAS links 00-04-08. Before the connection is closed, no other connection is allowed to be established over SAS links 00-04-08. That is, HBA considers SAS link 00 as busy, and if it wants to talk to disk E, it will send an open requests on, e.g., SAS link 01. After HBA is finished with disk D, it sends a close primitive on SAS link 00, and the close primitive is propagated through expanders B and C on SAS link 00->04->08. Disk D will respond with a close primitive, which is propagated back to HBA using the same path. After the handshaking process described above, the connection is successfully closed.
A SAS switch is one or multiple SAS expanders that facilitates SAS data transfer between multiple host HBAs/controllers and storage devices such as JBODs. A SAS switch provides 10 disaggregation enabling a rack storage pool to be segmented and dynamically allocated among servers. In certain applications, a large number of host HBAs and JBODs may be connected to a single SAS switch.
As stated above, the port count of a single physical SAS expander device limits the number of the SAS devices that can be connected to the SAS expander. Certain SAS expander applications, such as a top-of-rack SAS switch used to connect hosts and JBODs within rack, require a high port count and cannot be built on a single physical expander device. In order to implement a high port count in an expander device, a plurality of physical expander devices is grouped together and appears externally as a single cohesive SAS expander. However, existing method of grouping a plurality of expander devices into a single, cohesive SAS expander often includes using a number of homogenous self-managed expander devices which are synchronized over direct SAS connections. The synchronization connection may use direct SAS links among the SAS expander devices. The distributed SAS implementation in multiple SAS expander devices within a SAS switch brings challenges in SAS expander management, fail-over handling, enclosure management, etc. In the existing method the expander devices are also required to be assigned with identical SAS addresses so that wide port can be formed from PHYs from each of these expanders. This requires sophisticate design because the SAS standard does not allow an identical SAS address to be shared among multiple SAS devices within the same SAS domain.